Repository navigation
perf(writer): a node's integer min/max reuse the stats the competition computed (#583) - #700
Merged
Merged
Conversation
…petition computed (#583) CascadingCompressor.nodeStats scanned every cascaded integer array for its min and max, which the competition had often just computed for ArrayAndStats (integer stats always carry them, from the same kernel): Rust reads them off the array's stats cache. The array's ArrayAndStats is now built once in encodeWithCtx and shared by the competition and the node stats; the bounds come from it when it already computed its stats, and from a scan when it did not, never forcing the run/distinct/frequency counts for them. NodeStatsBoundsTest checks the reused bounds equal the scanned ones for every integer width, U64 past 2^63, nullable columns with nulls and an all-null one. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #583 (does not close it): the second min/max pass.
CascadingCompressor.nodeStatsscanned every cascaded integer array for its min and max, which the scheme competition had often just computed inArrayAndStats(integer stats always carry them, from the same kernel). Rust reads them off the array's stats cache. The array'sArrayAndStatsis now built once inencodeWithCtxand shared by the competition and the node stats; the bounds come from it when it already computed its stats, and from a scan when it did not (never forcing the run/distinct/frequency counts for them).Output is byte-identical by construction (same kernel, same input);
NodeStatsBoundsTestchecks reused vs scanned bounds for every integer width,U64past 2^63, nullable columns with nulls, an all-null and an empty one, andCascadingCompressorTestthat a real encode's root bounds equal a scan's for nullable and unsigned columns. The node-stats comparisons against vortex-jni inintegrationpass unchanged.Not measured: no benchmark figure, so no changelog entry.
🤖 Generated with Claude Code
https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV